The choice of a transmission rate for transmitting consecutive symbols or pulses in today's communication system is commonly based on so-called orthogonality considerations. Subsequent pulses are transmitted at a rate of with an intermediate or delay time T, where T is a time depending on the bandwidth of the pulse, meaning that the symbols can be transmitted at a rate of UT. The larger the bandwidth the smaller the delay time T. Subsequent pulses are called orthogonal to each other when the delay time T is chosen such that a transmitted symbol can be derived from only the corresponding received pulse; in other words there is no interference of other (nearby symbols), in the following also being referred to as Intersymbol Interference, ISI. The complete impulse response is then fulfilling the so-called Nyquist Intersymbol Interference, ISI, criterion; making a detection and estimation of the symbols at the receiver side particularly easy.
Orthogonality is not fundamentally necessary for communication. It has been found that a tighter symbol rate compared to the Nyquist rate may be possible without any severe transmission degradation. In order to further increase the data rates pulses might be packed tighter in time, which is equivalent to decreasing the delay time T between the Nyquist pulses. Such method is known as faster-than-Nyquist, FTN, signaling. As discussed above, such FTN signaling unfortunately introduces ISI as the pulses are no longer orthogonal. In other words, one symbol will hence affect the perception of other (nearby) symbols.
The first research in this area was carried out by B. Saltzberg, who published an article in 1968 titled “Intersymbol interference error bounds with application to ideal bandlimited signaling”, IEEE Transactions on Information Theory, vol. 14, no. 4, pp. 563-569, showing how the ISI affects the error probability. In 1975, J. E. Mazo published an article titled “Faster-Than-Nyquist signaling”, Bell System Technical Journal, vol. 54, no 8, pp. 1451-1462, indicating that the error probability in the FTN case may behave in some sense nice and would not worsen severely symbol detection due to ISI. In an article of J. Mazo and H. Landau, titled “On the minimum distance problem for faster-than-Nyquist signaling”, IEEE Transactions of Information Theory, vol. 34, no. 6, pp. 1420-1427, 1988; and in an article of D. Hajela, titled “On computing the minimum distance for faster-than-Nyquist signaling”, IEEE Transactions on Information Theory, vol. 36, no. 2, pp. 289-295, 1990; further proofs with respect to the Mazo assumptions have been provided. The cited articles however do not provide any receiver structure or method to cope with the ISI. In recent years it has been shown and numerically tested that a coding constellation for several different pulses (e.g. so-called sinc and root-raised-cosine pulses) may not induce a loss in minimum Euclidean distance if they are sent faster than the Nyquist ISI criterion allows. This is an indication that an optimal detector such as the Maximum Likelihood estimation (for equal input distribution of the symbols) should not suffer a loss in error rate even if signals are sent FTN (to a certain extent, depending on the pulse). The extent to which the constellation does not suffer any loss is called the Mazo limit. This notion has been extended even to the frequency domain. This means that different frequency channels may be packed tighter thus giving room for more channels.
The problem at hand when using FTN signaling is to provide an efficient coding/receiver structure to be able to perform reliable estimations of the symbols sent in the presence of ISI. Using a (state of the art) matched filter at the receiver, there exists a theoretical solution and an algorithm that solves the problem with minimum error estimation under the ISI that occurs when sending FTN. The Viterbi algorithm as it is called; is based on dynamic programming and is doing a maximum likelihood, ML, estimation of a so-called Hidden Markov Chain. In general performing an ML-estimation (which is optimal for equiprobal input) at the receiver (in a single-input single-output, SISO, channel) with ISI is a so-called non-deterministic polynomial-time, NP, -hard problem. The Viterbi algorithm is of exponential complexity and thus rendering it difficult or even impossible to use in a practical application where the number of symbols with ISI can be high.
FTN can for example be used to compensate for extra spectrum allocations due to non-ideal pulses used in implementation, which in essence translates to a capacity cost. Accordingly, 3GPP TS 25.104, V12.0.0 defines a so-called root-raised-cosine pulse with a roll-off factor of 22% to be used, meaning that the pulse has a frequency leakage of 22% percent compared to an ideal orthogonal pulse scheme, thus leading to an additional capacity of 22% in principal in the ideal case.